Papers with learning rate
Long Warm-up and Self-Training: Training Strategies of NICT-2 NMT System at WAT-2019 (D19-52)
Copied to clipboard
| Challenge: | NICT-2 neural machine translation system was presented at the 6th Workshop on Asian Translation (WAT-2019) |
| Approach: | They describe a NICT-2 neural machine translation system at the 6th Workshop on Asian Translation . they employ a long warm-up strategy and a self-training strategy that uses multiple back-translations generated by sampling to improve the translation quality. |
| Outcome: | The proposed system improves translation quality and learning rate by using the long warm-up and self-training strategies. |
Preserving Multilingual Quality While Tuning Query Encoder on English Only (2025.naacl-short)
Copied to clipboard
| Challenge: | Xiong et al., 2020, 2021b) and Dong eet., 2022) . Switching from one query encoder to another is easily feasible . |
| Approach: | They propose a general tuning technique that can be used to modify query representations for specific types of queries or domains while keeping precomputed and stored documents intact. |
| Outcome: | The proposed model preserves multilingual qualities and improves embedding qualities on different datasets. |
Meta-Learning based Deferred Optimisation for Sentiment and Emotion aware Multi-modal Dialogue Act Classification (2022.aacl-main)
Copied to clipboard
| Challenge: | Empirically, we show that the optimisation of multi-modal DAC, SA and ER tasks produces better results compared to its different counterparts. |
| Approach: | They propose a dual attention mechanism that integrates sentiment tags into a multi-modal conversational framework that integrate modal attentions and multiple loss optimization. |
| Outcome: | The proposed framework integrates sentiment tags for each utterance and learns generalized features across multiple tasks. |
Should I try multiple optimizers when fine-tuning a pre-trained Transformer for NLP tasks? Should I tune their hyperparameters? (2024.eacl-long)
Copied to clipboard
| Challenge: | NLP research has explored different neural model architectures and sizes, datasets, training objectives, and transfer-learning techniques. |
| Approach: | They propose to use a variant of Stochastic Gradient Descent (SGD) to select among numerous variants, often with minimal or no tuning of the optimizer’s hyperparameters. |
| Outcome: | Experiments with five GLUE datasets, two models and seven popular optimizers show that tuning just the learning rate is as good as tuning all the hyperparameters. |
MetaSLRCL: A Self-Adaptive Learning Rate and Curriculum Learning Based Framework for Few-Shot Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing few-shot text classification methods lack labeled data in many scenarios. |
| Approach: | They propose a meta learning framework that obtains different learning rates for different tasks and neural network layers to enable the meta learner to quickly adapt to new training data. |
| Outcome: | The proposed framework can obtain different learning rates for different tasks and neural network layers so as to enable the meta learner to quickly adapt to new tasks. |
Entropy Scheduling in Reinforcement Learning for Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | entropy in reinforcement learning functions analogously to the learning rate in LLMs. |
| Approach: | They propose an entropy scheduling system that optimizes different pre-set goals by controlling and scheduling entropicy at each step of the RL process. |
| Outcome: | The proposed method improves AIME2024 from 50.9 to 54.9 within 40 training steps. |
Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change (2020.acl-main)
Copied to clipboard
| Challenge: | Compared to previous studies, the performance of neural models is likely to be affected by the choice of hyper-parameters. |
| Approach: | They propose to automatically and dynamically determine batch sizes by accumulating gradients of mini-batches and performing an optimization step at just the time when the direction of gradients starts to fluctuate. |
| Outcome: | The proposed approach improves the Transformer model with a fixed 25k batch size by +0.73 and +0.82 BLEU respectively. |
Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | In order to achieve faster training we increase the mini-batch size and scale the learning rate accordingly. |
| Approach: | They propose a technique that delays gradient updates by increasing the mini-batch size to improve the model's convergence. |
| Outcome: | The proposed technique can train a shallow machine translation system 27% faster than an optimized baseline with negligible penalty in BLEU. |
Enhancing Language Model Hypernetworks with Restart: A Study on Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | a comprehensive investigation into optimization strategies for hypernetworks remains lacking. |
| Approach: | They propose restart optimization strategies to improve hypernetworks' performance for language models. |
| Outcome: | The proposed restart strategy improves hypernetworks' performance for language models, compared to conventional deep neural networks. |
Residual Prompt Tuning: improving prompt tuning with residual reparameterization (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt tuning is one of the most parameter-efficient approaches for parameter-effective tuning of pre-trained language models. |
| Approach: | They propose to reparameterize soft prompt embeddings using a shallow network with a residual connection and use it to tune prompt embeds P. |
| Outcome: | The proposed method outperforms prompt tuning on SuperGLUE, T5-Base and BERT-Bass models and can reduce the prompt length by 10 times without hurting performance. |
Understanding the Difficulty of Training Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Admin (Adaptive model initialization) is more stable, converges faster, and leads to better performance. |
| Approach: | They propose a model initialization algorithm to stabilize early training and unleash its full potential in the late stage. |
| Outcome: | The proposed model initialization method stabilizes early training and unleashes full potential in late stage. |
Evaluating Unsupervised Representation Learning for Detecting Stances of Fake News (2020.coling-main)
Copied to clipboard
| Challenge: | Using unsupervised representation learning, automated Fake News detection is a challenge for researchers. |
| Approach: | They examine pre-trained language models with respect to their performance on two Fake News related data sets. |
| Outcome: | The proposed models outperform the autoregression-based models on two Fake News related data sets. |
A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Models (2024.emnlp-main)
Copied to clipboard
Zhihao Wang, Shiyu Liu, Jianheng Huang, Wang Zheng, YiXuan Liao, Xiaoxin Chen, Junfeng Yao, Jinsong Su
| Challenge: | Version updates are an indispensable requirement for Large Language Models . a large learning rate in the first stage and a complete learning decay process are crucial for version updates of LLMs. |
| Approach: | They propose a learning rate path switching training paradigm for version updates of Large Language Models. |
| Outcome: | The proposed paradigm reduces training cost to 58% when training four versions of LLMs compared to PTFS and CPT . |
Hop, skip, jump to Convergence: Dynamics of Learning Rate Transitions for Improved Training of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Modern deep neural networks have achieved stateof-the-art performance across a wide range of machine learning tasks. |
| Approach: | They propose to switch the learning rate at a predetermined time during training to improve the performance of large language models. |
| Outcome: | The proposed model shows that switching the learning rate causes the loss curves to contract towards each other. |
GCML: Gradient Coherence Guided Meta-Learning for Cross-Domain Emerging Topic Rumor Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing domain adaptation rumor detection methods ignore the data generalization differences and rely on a large amount of unlabeled target domain samples to achieve domain adaptation. |
| Approach: | They propose a Gradient Coherence guided Meta-Learning approach for emerging topics rumor detection that selectively learns more "generalizable" tasks that are more beneficial in adapting to the target domain. |
| Outcome: | The proposed method outperforms baselines on real-world datasets and significantly outperformed traditional methods on the in-domain condition. |
Rethinking Table Instruction Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have overlooked the impact of hyperparameters on table understanding abilities . authors show that smaller learning rates and fewer training instances can enhance table understanding while preserving general capabilities. |
| Approach: | They propose a hyperparameter-based instruction-tuned model for table-related tasks that improves out-of-domain table understanding ability and general capabilities. |
| Outcome: | The proposed model outperforms existing models on table-related tasks while maintaining strong out-of-domain generalization and general capabilities. |
Training compute-optimal transformer encoder models (2025.emnlp-main)
Copied to clipboard
| Challenge: | OptiBERT is a family of compute-optimal BERT-style models that matches or surpasses leading baselines while training with dramatically less FLOPS. |
| Approach: | They propose to train OptiBERT models with a Masked Language Model objective . they train a family of compute-optimal BERT-style models that matches or surpasses leading baselines . |
| Outcome: | The proposed model matches or surpasses leading baselines on GLUE and MTEB while training with dramatically less FLOPS. |
How to Set the Learning Rate for Large-Scale Pre-training? (2026.findings-acl)
Copied to clipboard
| Challenge: | Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. |
| Approach: | They propose a Fitting Paradigm and a Transfer Paradigme to investigate fit and transfer . they propose scalability and elucidate the reasons why module-wise parameter tuning underperforms . |
| Outcome: | The proposed model reduces the search complexity by reducing the search cost by lowering the search factor. |